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ABSTRACT * : 

v / A program evaluation unit was established in the 
Louisiana Departtaent of Education. A unit capable of measuring and. 
assuring the effectiveness of several programs and projects either 
operating or proposed was desired. The Department responded to 
/requests to implement program evaluations. Many requests were from 
program administrators who needed .formative afnd summattive 
evaluations. It was*,nece t ssary, for the superintendent and his cabinet; 
to develop and appr'oye a "policy statement relative to program 
evaluation. Three key points include: program evaluation .was separate 
). tr t om program administratis^ results were reported, tq 'the 

superintendent, M)i^jcabinet, and program administrators; and , 
financial support was provided through the prpgram budget. The 
director^ of the Program Evaluation Bureau reported to the associate 
' superintendent of Research and Development. The organizational 

structure was.jconduciv* to the drderred growth of p?<*gram evaluation 
as it assumed additional responsibility; supported p'rogram evaluati 
with data and research bases, as well as computing and dissemination 
, power; and provided itr with acdessjto the decision makers. The unit* 
.i^ was staffed with professionals trained and experienced in program 
evaluation* It was«established in 1980 and seems 'to have a bright 
professional, as well as funding, .future*; (DWHlL 
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THE MANAGEMENT OF AN EVALUATION' UNIT 
' ^ . • ' " IN A STATE EDUCATION AGENCY 

' Hugh I. Peck 
Louisiana Department of Education •* 

^ The issues facing education decision -makers today are becoming 

rr 1 * more complex than any time in history. DSfcision-makers^fwn the 

Q highest levels of Federal government through state and local .level 

government and individual school principals are required to make 
decisions on complex issues within the equally • complicated 
circumstances of public education. Decision-maker^ have the rigfK and 
obligation | to look to education, program evaluators, /as well as 
• researchers, for accurate and timely information on which to make 
those decisions. ' Program evaluators must be able to focus questipns 
• and design evaluations that will respond to tfie informational needs of a 
wide yariety of users. Program evaluators must have a .large 
repertoire of research methods appropriate to an extensive' variety of 
problems. 

' * The tasks* of program .evaluation today call upon such social 

• 1 . 
. science research methods as quantitative data analysis, questionnaire 

• results, , secondary data * analyses, cost benefit and effectiveness 

analyses, standardized tests, experimental designs, unobtrusive 

measures, participant observation, and in-depth interviewing. It is 

the responsibility of the evaluator to design evaluations that include all 

• data, . given the constraints of resources and time, that will shed light 

on evaluation questions such as: 

^ ' • Was the program (project or material) implemented as it was 

designed? 
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»• Whart effect did it hayei * ' ^ X 

• Did it make any difference? 

• What did it cost? . * . . ' 
Was it worth it? J •< 

• Can- it be disseminated effectively? , 

It is within this context that the Louisiana Department ,of 
Education; the SEA, addressed tire* need for a program evaluation unit 
within the agency and the need for extensive support of that unit. 
The^ Department's and the State Board's concern about the quality of 
the education evaluation efforts under way in our locaj city and parish 
school distHcts : (LEAs) gave, impetus to the establishment of an 
evaluation unit. We knew that hundreds of thousands of dollars were 
being spent for what was being labej^d evaluation without any 
assurance that the evaluators knew how to evaluate or that the 
evaluations provided any useful information.; . At the same time, the 
Department was' being drawn more and more into a position of requiring 
good professional evaluations of the numerous new programs it was 
•charged with evaluating. Legislators were beginning to require that 
evaluation reports accompany * renevyed* requests for program funding 
and that newly' funded programs be evaluated routinely. The«State 
Board members were qualifying their support of, certain programs 
pending evaluations. The Chief State School Officer was requesting-- 
no, demanding—evaluation results in his decision-making. His cabinet 
members saw the writing on ^the wait and also began to seek systematic 
evaluations of tjieir programs and program areas. . > 

■ ' ' ' ' i. ^ * " ' 

* . The conceptual framework for an evaluation unit in the Louisiana 

• • '* 
Department of. Education pentered around the role .of the SEA in 
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establishing and maintaining programs . of quality" assurance in all major 

endeavors of the Department and in^ our outreach to the LEAs. The 

Department wanted, an evaluation unit capable of measpring and 

assuring the effectiveness of a v?riety "of programs "and projects that 

were either operating or proposed. Further the Department felt the 

* «. " • 

need for an evaluation research effort that could provide information to 

program decision-makers on an ^ongoing basis. Requirements for the 

V • ■ * 

Department to implement program evaluations came from a variety of • 

sources. It is a cflfpmon practice for the Legislature to attach' 

evaluation mandates to , 'certain allocations for education programs such 

as the State's m upcoming Compensatory/Remedial Education program. 

The State Board of Elementary and Secondary Education often demands 

evaluative information on*programs under ' their purview. The thief 

state school officer, otf^nunrterous occasions,* requests evaluations of. 

* • » < 

certain Department efforts. (It should be noted* here threat another* 

ynit — Management and Program Analysis--iF responsible for the 

assessment of management efforts within the Department, and proposed 
V 

efforts before funding and implementation:) 

v - . < 

Requestsxfor program evaluations come most often, h&wever, from? 

the program managers or administrators. They see the neecjh for both 

formative and summative evaluations. * ' j \ / , 

The Department h'ad not moved far into %he - establishment* of a 

program evaluation unit before it became xieces^ary for the' 

superintendent and his cabinet to develop and approve a policy 

statement relative to program evaluation*. Figure I is I copy of that 

policy statement. The Statement contains* three key - points: 

(I) program evaluation' is separate ;frbm,, program administration; 




1 FIGURE 1: PROGRAM EVALUATION POJLICY STATEMENT 
\' ' * m : STATE OF LOUISIANA 

DEPARTMENT OF EDUCATION 

' TELEPHONE NO: 1400-272*9872 
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August 22, 1980 
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Dr. C. E. Thompson, Deputy Superintendent for Education 



SUBJECT: Operating Procedures^ for the Bureau of ^Evaluation \ 
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The Bureau • of Evaluation^ while being a part of and Integral t\ the Department, of Education, will provide 
evaluation services independent of program administration withib the Department of Education. The Bureau 
of Evaluation is positioned in the Office of Research and Development, and evaluation staff report to the. 
Director of the Bureau who, in turn; reports to the Assistant Superintendent for Research and Development, 
flevesfheless, because of the nature of prpgram evaluation, the Bureau will be working closely with other 
bureaus and' programs within the Department * 4 * " 

In order to clarify the relationship of the Bureau of Evaluation with the programs begin evaluated, the 
j following procedures have been adopted by the Department : * 

I . • / , • 

»' • The Bureau of Evaluation will through regular budgeting procedures and legislative 

appropriation, have a core budget, which provides for D'the overall administration . 
of the Bureau, 2) the administration^ and implementation of .the. quality control 
functions of the Bureau, and 3) the ongoing evaluation of projects, programs and 
materials administered by the Department of Education. 

V ^-jpe Bureau of Evaluation will negotiate with individual divisions, bureaus and « 

programs within the 'Department of Education HEor an appropriate evaluation of m - • «) 

specific projects,' programs akd materials, These individual evaluationjactiyi- 
- , ties will be supported within* the* budget of the respective project or program 

/ in a subcontract arrangement with ^he Bureau of Evaluation. The evaluation , 
of a project, program or- material will be negotiated at the beginning of the 

• planning process. Upon initiation of the project or'^rogram, evaluation funds 
'will be transferred to the Bureau of Evaluation cost center. 

/ * > ' 

( • The Bureau of EvaluatiotwwiU prpvitfe a written design document for all evalua- 
tions for which it is 1 responsible. The design document negotiated with the 
% e project or--program staff will contain, among other things, all evaluation and 
cpolicy questions to be addressed by the evaluation, the conceptual framework, 
populationr data fcase, analyses' for the evaluation and the budget requirements. 

• The Bureau of Evaluation will pr«P**« appropriate written reports of the 
evaluation. Generally, reports Will-be of three types:- a technical volume, 
an executive summary, and an* abstract. As standard procedure,' copies will, - «• 

' be submitted tqjhe Superintendent:* the Deputy Superintendent, the Assistant 

Superintendent fir Research and Development, the assistant superintendents 
and .bureau director administratively responsible for the evsluated project, 
program or material, and the project /program administrator. - 

• # « 

. ''Further, it is the posture of the Department of Education that generafly evaluations 'of programs' administered 
;by t&t Department are to be conducted by the Bureau of Evaluation. # 
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(2) program evaluation results are reported to the superintendent, his 

♦ % -» 1 
cabinet, an'd program administrators; and (3) program evaluation is 

supported financially -through the program* budget with funds being 

transferred to the, Bureau of Evaluation once the evaluation begins. 

The need for quality assurance at both SEA and ievels led to 

the adoption by the State Board v>f Elementary and Secondary 

Education of the Joint Committee on Education Evaluation's Standards 

* * * • 

*for Evaluation of Educational v Programs/ Projects, and Materials . The 

statement of evaluation -standards had. been published in 1980 only 

weeks before its adoption. Therefore, the Department and Board were 

working from drafts and "bluelines" during the adoption hearings. It 

was a'first, and as far.as we know, no othej state has yet adopted the * 

standards for statewide application. For Louisiana they are a major' 

part of our overall philosophy of evaluation. r Their adoption and 

9 * * - - • » 

. tr \ * 

# application is a chronicle of its own, and the story was told during .the 
I982~AERA Symposiurft, "The Impact of a Quality Assurance Model on 

•Prpgram Evaluation in Louisiana.";-* • ~ 



' Often thfe pja^n^jf;jbf "any program unit within a larger overall 

.agency.Jis^ pot r g^Ven seYidus, enough consrdf ration. The program 

^&A<*? : ' % . . ■ ' 

'<£vdfuation unit-needs^ to exist where Jt can be nurtured, supported, * 
administered, and expanded through a maturing . process that may be' 
troubled by policy /o personnel, financial, and role problems. 

To understand hoyv the firganizattonal structure of the Louisiana 
SEA {positions ancj uses the Bureau of Program Evaluation, it is 
necessary to\ee tu£> .charts of the organizational structure of the 
Department. .Figure 2 depicts the administrative structure at- the • 
cabinet leyil. Note that the Offipe'pf KesearclAand'Devefopment is led 



FIGURE 2 



ADMINISTRATIVE STRUCTURE 

PEOPLE OF LOUISIANA' 
GOVERNOR 
LEGISLATURE 
STATE SUPERINTENDENT OF EDUCATION 



MANAGEMENT AND FINANCE* 



•Accounting 
rfodltlng 0 
•budgeting 

•Operational Support Services 
•Personnel and Training 
•Purchasing and Stock Control 
-State Financing of Schools 
•Management.. 
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DEPUTY SUPERINTENDENT 

Executive Staff * 
Public Information 

RESEARCH *ANO DEVELOPMENT 
•Acceptability 
•Development 
•Evaluation 

•Management Information Systems 
•Research 



AUXILIARY* PROGRAMS-SPECIAL PROJECTS 



iCETA 

Food and Nutrition Services 
Veterans Education 
New Means Regional Center 



SPECIAL EDUCATION 



•Psychological and School * 

Social Work 
•Special Education Developmental 

Programs and Planning 
-Special Education LI afsons 

.and Program Specialists 
-Special Education Program 

Services 



VOCATIONAL EDUCATION 



• Administrative Services I Special Needs 
— Proprietary Schools 

• Secondary Programs 

• Trade, atf ^Industrial Education 

• Vocation AtEducatlon Coordinating Unit 



EDUCATIONAL SUPPORT PROGRAMS 

•Materials of Instruction 
•Migrant Education 
•School Transportation 
•Title I. ■ 
•Title IV 



ACAOEKIC PROGRAMS 

•Academic Support 
•Adult /Community 
• .Education i 
•Continuing Education 
•Curriculum* Inservlce >and 

\ Staff Development 
•Elementary Education/ . 

Reading 
•Higher Education/ 

Teacher Certification 
•Secondary Education 
/•Student Services * 
•Safety Education 



oy an associate superintendent and has direct access to, the chief state 

• * * * 

schpol officer. This, it seemed to us, was a key*factqr, if program 

evaluation were to affect decision-making at the highest levels. Figure 

3 is an organizational chart of the Office of Research and Developments 

i • 

4 - \ 

Anqther key to our-pi^graifi evaluation "effort was that it be located 

close to and within tlie same Office as jts major support systems. The 

research unit houses an extensive education library, as welj as units 

for informati6n and analytical services. 

The Bureau of Accountability has among its responsibilities both 

TpubJic and nonpublic testing, which are^ important- sources "for 

evaluation data. The* Bureau of Management Information Systems 

directs the Educational Computing Network of Louisiana, and is a 

support source for data and word processing, both vitaf to program 

evaluation. Finally, the Bureau of Development not only is- on the 

cuttin^ecjcje of all educational Innovation in Louisiana, but also houses 

our Louisiana Dissemination Network, an NIE funded program th;at 

provides linkage among the SEA, the LEAs, and institutions of higher 

-education. This sytems provides the conduit for the statewide 

dissemination of results, programs, practices, and resources. 

The director of the Program Evaluation Bureau is responsible to 

the associate "superintendent of Besparch and Development as are the 

directors . of Accountability, Research, Development and Management 

Information Systems (MIS). J+ie organizational structure of the 

Bureui of Evaluation itself had to be designed* around the functions 

I m * ' 

that were to\be addressed by the> Bureau, taking jnto consideration 

personnel and staffing as well as financial -constraints. Figure 4 is the 

organizational chart for the Bureau of Evaluaticyj. An examination of* 
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FIGURE 3* 
ORGANIZATIONAL CHART 



^OFFICE OF RESEARCH AND DEVELOPMENT 
FUNCTIONAL ROLE AND JtESPONSIBIL^TY* 



Associate Superintendent 



Director, 

BESE/Nlgher Ed. Liaison. 
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Development 
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Director, 
Research 




Director, 
Evaluation 




Director, 

Management Information 
Systems 
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• Student Assessment 
Personnel Evaluation 

(LEA ft SEA) 
Nonpublic School Testing 
tylnlmum Competency Resting 



Program Accreditation 

and Review 
Dissemination ' 
Program DevelopmenxV 

Services 
Technical Services 



Applied Research 
Legislative Assistance 
Legal- Research 
Resesrch Clbrary ^ 
Publications * ./ 
Forecasting Analysis 
Stsndsrdlsed Financial 
Accounting 
. I 



Program Evaluation . 
Project Evaluation 
Process Evaluation 
Evaluation Standards 
Evaluator Registry 



Educational Computing 
Network 7 
Systems Development 
Date Element Dictionary 
Data Acquisition Plsn 
Word Processing Services 



FIGURE 4 * 
ORGANIZATIONAL ' CHART 
BUREAU OF EVALUATION 
Offlc6 of Research and Development 



Director 



• Educational Computer Technician I, 
L Administrative Specialist 



Program Evaluation 4 

• Section Chief 

• Administrative Officer 

• Administrative Officer 

• Administrative Officer 

• Administrative Specialist 



Evaluation Studies 

• Administrative Officer 

• Administrative Officer 

• Administrative Specialist 
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•the chart, reveals that the structure follows carefully the philosophy , 
and policy areas, we have discussed .earlier In this paper. A sub-unit 
6f the JBureau is responsible for the evaluation of programs that the 

Department and others are implementing. There is also a sub-unit for 

\ « 

quality control * and % .assurance tthat assists the " Department's - 
Certification, unit in the certification of evaluators and implements 

trailing* in the" Joint Committee Standards, a function the Department t 

** 

soon hopes to turri over to the Universities. This sub-unit is .also 
responsible for the application of the standards to state, and local 

education evaluation efforts.^ The third unit, is responsible for 

♦ y 
evaluation research* into program and policy, areas, of concern to 

4 / . - * 

education administrators in the Department, where systematically 
collected 'data is needed on a day-to-day basis, so that decision^e*^ be 
.made. 

The Louisiana Department of • Education . has , developed an 
organizational structure that (I) is conducive to the ordered growth of 
program evaluation ° as it assumes more and more responsibility; 
(2) supports prog rafti evaluation with the necessary data and research 
bases, as Well a$ computing and dissemination power; and (3) provides 
it with direfct access to the decision-makers. 

Jhe first professional employed* in April, 1980 for the Bureau was 

*• %■ '* 

the director, who not only assumed other management responsibilities, 
but ajso financial management as well. Building a financial base for 
program evaluation* has been a deliberate proqess that has mixed state 

-f. - - - - - ^ — ~ *" 

and Federal funds « with: funds generated through contract-lil^ 

arrangements >vith "clients within the' Department. The 1980-81 budget 

f. - * v * • * . 

was for the most part adjusted frqifi within the Office of Research .and 



Development, butjalsb included project jFjyyids generated from clients 
.within the Department, such \s Academic Programs ^and Special 
Education* The structure of the overall budget provides both a core 
budget sefficierit to fund five or six senior professionals, witfy other 
necessary support and a project budget with fundsi&$rom contract 
evaluations as necessary to do the various evaluation tasks required or 
requested by the Department. 

The projected 1981-82 core ^budget was $364,500 of state 
disdretiona^yj Title iy-C funds which, by $he way, was quickly 

reduced by Federal budget cuts to ^$167,000; the qpre staff was cut 

h ■ , 

from eight to five. An additional $200,000 was ; generated' from 

contractual agreements for a totaJ 1981-82 budget of $367,000. Realizing 

that hopes for Federal funding in 1982-83 are nil, the Bureau projected 
% 

a*l982-83 core budget of $425,000 of state fuijids, plus a project budget 
of another $200,000 of r funds generated through interagency 

& * 

agreements. These funds were incorporated . into the Departments 

■ p • 

overall budget through the Off ice of Research and Development. 

At the Louisiana 'School „ Boards' Association's annual meeting in 
Louisiana, the Governor announced his support for the program 
evaluation unit. It would seem the Bureau of Evaluation has excellent 
prospects for full funding in. 1982-83. Obviously, wi^out financial 
resources, the Bureau would face tough timfes; However, things* look 
very promising for its future. Without the total commitment of the 
Department, and, especially the state superintendent, resources would 
hot be available for : its future. 

Management of any agency unit and its success or failure is 
basically a*. personnel challenge. All of education fails or succeeds, 



dependent upon' the people who a,re employed to carry out those 
education A and evaluation responsibilities. The first employee, the 

• ' * « » 

director, a Ph.D* f*rfem D&ke University with 10 years of h&nds-on 
evaluation experience and five years of teaching, presented to the 
Associate Superintendent a matrix staffing plan that proposed using 

trained, experienced evaluators like utility infielders. . This plan. 

# * 
♦ # 

assigned taajor roles to each staff member but capitalized on individual 
talents across functional needs. 

Since J Louisiana has adopted certification requirements for 
evaluators as a part of its program evaluation efforts (as* we hive for 
teachers imd # p rirvoipals) , all evaluation professionals on the SEA Staff 
had to be among the first group to receive Board of Elementary and 
Secondary Education certification. -The story of certification of 
evaluator? is not necessarily germane to this management analysis, but 
is a part of the symposium already mentioned in this paper. It is 
estimated that the evaluation unit will be staffed with ^10 to 12 
professionals trained and experienced in program evaluation, quality 
assurance, evaluator training, and research on the evaluation process 
by next year this time. 

Additional personnel plans call for the quarterly evaluation of the 
director by the associate superintendent, as -well as the quarterly 
evaluation of each staff member by the superviscto^p whom he/she is 
responsible. „ These evaluations are based ori performance objectives as 
well as employee characteristic*. 4 

In times of : shrTnkfng i budgets, diminishing student - rolls, and 
^reductions in force, the Louisiana Department of Education . has seen 
the education and economic value of a program evaluation unit and has, 



With bold and careful plans, organized, funded and put in place a>, > N 
dynamic unit responsible for statewide efforts In program evaluation. < 
Though the unit has existed onlV since April I, 1980, i* has grown to a, 
reasonable size and its funding Nature, as well as its professional 
future, looks bVight. 
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